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ABSTRACT 

Internet-based learning is increasingly applied in medical education, but its effect in the field of public health training is 
still unclear. This meta-analysis was undertaken to explore the impact of Internet-based learning on 
students/professionals’ knowledge of public health compared with no intervention and with traditional face-to-face (FTF) 
formats. Two reviewers independently searched Medline, Web of Science, ProQuest, Google scholar, ERIC and Elsevier 
databases for relevant studies between 1st January, 1990 and 30th December, 2016. Studies in English language 
providing information on educational outcomes after Internet-based training in public health courses compared with 
no-intervention or a pre-intervention assessment, or with FTF control group were retrieved, reviewed, and assessed 
according to the established inclusion/exclusion criteria in the current study. There were 16 eligible studies with 1183 
participants in total. Heterogeneity in results was detected across studies. A random effects model was used to pool effect 
sizes for knowledge outcomes. The pooled effect size (standardized mean difference, SMD) in comparison to no 
intervention was 1.92 (95% Cl: 1.05 to 2.78; P<0.0001), favoring Internet-based interventions. Compared with FTF 
formats, the pooled effect size was 0.39 (95% Cl: -0.06 to 0.83; P=0.09). The study suggested that Internet-based 
learning was superior to no-intervention in improving students/professionals’ public health knowledge. Compared with 
traditional FTF formats. Internet-based learning showed a similar effect. 
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1. INTRODUCTION 

Internet-based learning, which has been defined as using the Internet to deliver and access learning materials, 
to interact with peers and instructors and to enhance knowledge and performance (Wutoh et al. 2004), is 
increasingly applied in medical education over the past few decades. Compared to traditional learning 
approach, Internet-based learning has advantages in satisfying an extensive range of learning needs and 
permitting flexible learning transcending time and space (Cook et al. 2010; Greenhalgh 2001). Its growing 
spread has led to discussion, research, and debate over the effect of Internet-based learning compared with 
traditional face-to-face (FTF) instruction and no-intervention (Choules 2007; Greenhalgh 2001; 
Hemans-Henry et al. 2012). 

Quantitatively summarizing these evidences could increase statistical power and is more convincible to 
inform educators and learners about the extent to which the Internet-based learning is effective. Currently, 
the reviews concerning the effect of Internet-based learning in medical education have mostly focused in the 
field of clinical medicine or basic medicine (Cook et al. 2008; Lahti et al. 2013; Lewis 2003). Cook DA and 
his colleagues reported positive results in their meta-analysis comparing Internet-based intervention to 
no-intervention in clinical medicine (Cook et al. 2008). Lahti M et al. reported that Internet-based learning is 
as good as traditional learning methods in nursing courses in a review (Lahti et al. 2013). 
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However, so far there is no quantitative pooling to estimate the effect of Internet-based learning in the 
field of public health. As Internet-based learning is a new field, interventions are not yet clear. According to 
the previous meta-analysis (Cook et al. 2008; Lahti et al. 2013), we expect large heterogeneity of studies, 
especially of intervention. Two meta-analyses were conducted in the present study, the first exploring 
Internet-based compared with no intervention or pre-intervention assessment, and the second summarizing 
the studies comparing Internet-based and traditional FTF instructional methods. 


2. MAIN RESULTS 


2.1 Data Sources 

To identify the relevant studies, Medline, Web of Science, ProQuest, Google scholar, ERIC and Elsevier 
databases were searched. Key search terms included delivery concepts (such as “Internet, OR Web OR 
computer OR distance OR online” AND “learning OR instruction OR education OR training”), study design 
concepts (such as “evaluat* OR compar* OR pretest OR effect*”< * is a truncation symbol for searching. 
For instance, evaluat* would retrieve entries containing the words: evaluate, evaluation, or evaluative, etc>), 
and educational topics “public health/education [mesh]”. To complement the search strategies, keyword 
searching of Google scholar was also conducted. The reference lists of retrieved articles were reviewed to 
identify any additional publication. More details on the search strategies were available upon request from 
the authors. 

2.2 Search Results 

The search strategy identified 709 studies from the databases. A review of the reference lists of the retrieved 
studies identified additional 13 potentially relevant articles. Out of the 722 publications, 109 potentially 
eligible articles were read in full text for further assessment. From these, 22 were considered appropriate for 
inclusion. After that, six studies were excluded due to insufficient data for coding quantitative outcomes. 
Thus a total of 16 studies were included in the quantitative analyses, among which four studies compared the 
Internet-based learning with both no intervention and traditional FTF instruction (Abdelhai et al. 2012; 
Hugenholtz et al. 2008; McGready and Brookmeyer 2013; Smits et al. 2012). Therefore 12 studies were 
contributed to meta-analysis of no-intervention comparison (Abdelhai et al. 2012; Aggarwal et al. 2011; 
Chung et al. 2004; Curioso et al. 2008; Farel et al. 2001; Hugenholtz et al. 2008; McGready and Brookmeyer 
2013; Ried 2010; Sears et al. 2008; Smits et al. 2012; Steckler et al. 2001; Zahner 2006), and 8 contributed to 
meta-analysis of traditional FTF instruction comparison (Abdelhai et al. 2012; Campbell et al. 2008; Evans et 
al. 2007; Fayram and Anderko 2009; Hugenholtz et al. 2008; McGready and Brookmeyer 2013; Rose et al. 
2000; Smits et al. 2012). The search flow was shown in figure 1. 

2.3 Study Characteristics 

The included 16 studies were published between 2000 and 2016. Table 1 summarizes key characteristics of 
the included studies. The sample size of the included studies varied from 13 to 138 participants, and the total 
number was 1183. Besides one study conducted in Egypt (Abdelhai et al. 2012), one in India (Aggarwal et al. 
2011), the remaining 14 studies were in Europe and North America, including two in Netherlands 
(Hugenholtz et al. 2008; Smits et al. 2012), one in the United Kingdom (Campbell et al. 2008), one in 
Canada (Sears et al. 2008), and ten in the United States (Chung et al. 2004; Curioso et al. 2008; Evans et al. 
2007; Farel et al. 2001; Fayram and Anderko 2009; McGready and Brookmeyer 2013; Ried 2010; Rose et al. 
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722 studies identified and 
screened for retrieval 
709 database search 
13 article reference lists 


613 excluded 

a. 32 duplicates 

b. 278 not original research (i.e. review) 

*j c. 229 the main themes not related to public health education 

d. 17 instruction not offered predominantly via the Internet 

e. 57 not health professional and/or student learners (i.e. 
client/patient) 


87 excluded 

a. 11 meeting abstracts 

▼ b. 6 the main themes not related to public health education 

109 of articles retrieved | c. 17 instruction not offered predominantly via the Internet 

for eligibility assessment | d. 48 no quantitative or qualitative comparison data (i.e. 

I 1 posttest-only group) 

e. 5 not health professional and/or student learner (i.e. 
client/patient) 


▼ 

22 considered 
appropriate for inclusion 


v _I_ 

8 of articles included in traditional 12 of articles included in no¬ 
education control meta-analysis intervention control meta-analysis 


Figure 1. How diagram 

2000; Steckler et al. 2001; Zahner 2006). Besides biostatistics and epidemiology courses (6/16), topics 
such as occupational health, community health, reproductive health, tobacco and public health, and 
quantitative and qualitative research methods were also addressed. In addition, review of the selected articles 
revealed big variations in the Internet-based education modalities, including tutorial, videoconference, case 
study and online discussion with peers and/or instructors, et al. 


6 excluded for insufficient data 
for quantitative outcomes 


2.4 Quantitative Data Synthesis of No-Intervention Comparison 

Twelve studies were included in the analysis (Fig 2). The pooled effect size of 1.92 (95% Cl: 1.05 to 2.78, 
P<0.0001) significantly reflected a large effect according to Cohen’s criteria (Cohen 1992), which suggested 
that Internet-based learning has a substantial benefit on learners’ knowledge compared with no-intervention. 
However, there were large inconsistency across studies (I 2 =97%), and individual effect sizes ranged from 
0.10 to 5.41. The test of funnel plot asymmetry indicated publication bias among studies (Egger’s test 
P=0.003). We executed sensitivity analyses by sequential omission of individual studies to reflect the 
influence of the individual data on the pooled effect size and evaluate the stability of the findings. The results 
demonstrated that the sensitivity analyses would not considerably affect the conclusion. 

Subgroup analyses were further performed and categorized by study design, methodological quality scale, 
and instructional design aspects to explore the sources of inconsistency. The effect sizes showed consistently 
positive number (favoring Internet-based learning) when conducting each subgroup analysis respectively, 
although some of the results became statistically insignificant (undergraduate degree courses, duration less 
than 1 week, absence of exercise, and with single access to course materials)(Table 2). 
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Table 1. Characteristics of the included study 



No-intervention control 

Traditional instruction control 

Study characteristic 

No. of studies 

No. of 
participants 

No. of studies 

No. of participants 

All studies 

12 

951 

8 

865 

Design 

pre-posttest, 2 group 

4 

633 

4 

633 

posttest, 2 group 

0 

0 

4 

232 

pre-post, 1 group 

8 

318 

0 

0 

Quality Newcastle scale 

= 4 points 

9 

558 

5 

493 

= 3 points 

3 

393 

3 

372 

Country 

USA 

7 

368 

4 

266 

Netherlands 

2 

200 

2 

200 

Egypt 

1 

295 

1 

295 

Canada 

1 

59 

0 

0 

UK 

0 

0 

1 

104 

India 

1 

29 

0 

0 

Degree 

undergraduate 

3 

527 

3 

474 

graduate 

3 

225 

5 

391 

not mentioned 

6 

199 

0 

0 

Object 

student 

4 

665 

4 

612 

professionals 

8 

286 

4 

253 

Duration 

' 1 week 

7 

625 

5 

620 

< 1 week 

3 

259 

2 

200 

not mentioned 

2 

67 

1 

45 

Exercise 

present 

9 

766 

5 

582 

absent 

3 

185 

3 

283 

Discussion 

present 

5 

536 

4 

569 

absent 

7 

415 

4 

296 


Studv or Subarouo 

Weiaht 

Std. Mean Difference 

IV. Random. 95% Cl 


Std. Mean 

IV. Rando 

Difference 

m. 95% Cl 

McGready J 2013 

8.7% 

0.27 [-0.06, 0.60] 





Smits P 2012 

8.7% 

0.10 [-0.25, 0.44] 





Aggarwal R 2011 

8.5% 

0.36 [-0.16, 0.88] 





Ried L.D2010 

8.5% 

5.17(4.60, 5.74] 





Hugenholtz N.R 2008 

8.5% 

1.43 [0.91, 1.94] 





Curioso W.H 2008 

8.6% 

0.61 [0.16, 1.07] 



■ 


Sears K.E 2008 

8.2% 

5.41 [4.62, 6.19] 





Zabner S.J 2006 

7.6% 

3.02 [1.85, 4.20] 





Chung S 2004 

8.5% 

0.34 [-0.17, 0.84] 





Farel A 2001 

8.5% 

0.98 [0.42, 1.54] 





Abdelhai R 2012 

8.7% 

1.53 [1.26, 1.79] 





Steckler A 2001 

7.0% 

4.65(3.20, 6.10] 




- 

Total (95% Cl) 

100.0% 

1.92 [1.05,2.78] 





Heterogeneity: Tau 2 = 

2.21; Chi 2 

= 430.63, df = 11 (P < 0.00001); I 2 = 97% 

-4 

-2 

2 

4 

Test for overall effect: Z = 4.35 (P < 0.0001) 

Favours No-intervention 

Favours Internet-based 


Figure 2. Forest plot for no-intervention control meta-analysis 
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Table 2. Pooled effect sizes (standard mean difference, SMD) for subgroup meta-analyses of Internet-based learning vs. 

no-intervention 


Subgroup 

No. of 
studies 

No. of 
participants 

SMD (95%CI) 

12 

p-value 

All studies 

12 

951 

1.92(1.05, 2.78) 

97% 

<.0001 

Design 

Pre-posttest, 2 group 

4 

633 

0.82(0.05, 1.60) 

95 % 

.05 

Pre-posttest, 1 group 

8 

318 

2.53(1.01,4.05) 

98% 


Quality Newcastle-scale 

^4 points 

9 

558 

1.74 (0.68, 2.80) 

97% 

.51 

^3 points 

3 

393 

2.48 (0.53, 4.42) 

97% 


Degree 

undergraduate 

3 

527 

2.25 (-0.01,4.51) 

99% 

.93 

graduate 

3 

225 

1.91 (0.35, 3.48) 

95% 


not mentioned 

6 

199 

1.74 (0.43, 3.05) 

97% 


Participant 

student 

4 

665 

1.75 (0.11, 3.39) 

98.9% 

.79 

professionals 

8 

286 

2.01 (0.93, 3.09) 

96% 


Duration 

£ 1 week 

7 

625 

2.12(0.87, 3.36) 

98% 

.07 

< 1 week 

3 

259 

2.29 (-0.29, 4.87) 

99% 


not mentioned 

2 

67 

0.76(0.41, 1.11) 

0% 


Exercise 

present 

9 

766 

2.43(1.31, 3.55) 

98% 

.002 

absent 

3 

185 

0.44 (-0.06, 0.95) 

71% 


Discussion 

present 

5 

536 

2.33 (0.94, 3.73) 

98% 

.47 

absent 

7 

415 

1.64 (0.37, 2.90) 

98% 


Repetition 

persistent access 

8 

663 

1.99 (0.93, 3.05) 

97% 

.86 

single instance 

4 

288 

1.79 (-0.02, 3.61) 

98% 



2.5 Quantitative Data Synthesis of Traditional FTF Instruction Comparison 

Eight traditional FTF instruction controlled studies were included in the present analysis (Figure 3). Although 
the random effect size showed some improvement associated with Internet-based learning compared to 
traditional FTF format, it was not statistically significant (SMD: 0.39; 95% Cl: -0.06 to 0.83; P=0.09). 
Individual effect sizes ranged from-0.32 to 1.30. The heterogeneity amongst studies was large (I 2 =88%), but 
the test of funnel plot asymmetry indicated no evidence of publication bias (Egger’s test P=0.665). 


Std. Mean Difference 


Std. Mean Difference 


Studv or SubarouD 

Weiaht 

IV. Random. 95% Cl 

IV. Randc 

m. 95% Cl 

McGready J 2013 

13.9% 

0.11 [-0.22, 0.44] 



Smits P 2012 

13.8% 

0.03 [-0.32, 0.38] 



Abdelhai R 2012 

14.4% 

1.30 [1.05, 1.55] 



Fayram E.S 2009 

9.9% 

0.67 [-0.15, 1.49] 



Hugenholtz N.R 2008 

12.9% 

0.08 [-0.38, 0.55] 



Campbell M 2008 

13.4% 

0.63(0.22, 1.03] 


■ 

Evans S 2007 

10.4% 

-0.32 [-1.08, 0.43] 



Rose M.A 2000 

11.2% 

0.47 [-0.20, 1.14] 



Total (95% Cl) 

100.0% 

n 34 • P.hi 2 = 

0.39 [-0.06, 0.83] 

68 6Q rif = 7 ip < n nnnm v p = aa% - 

-1- 

-1-1- 


Test for overall effect: Z = 1.71 (P = 0.09) 


-4 -2 0 2 4 

Favours Traditional Favours Internet-based 


Figure 3. Forest plot for traditional F2F instruction control meta-analysis 
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Table 3. Pooled effect sizes for subgroup meta-analyses of Internet-based learning vs. F2F-intervention 


Subgroup 

No. of studies 

No. of participants 

SMD (95%CI) 

12 

p-value 

All studies 

8 

865 

0.36 (-0.10,0.83) 

89% 

.12 

Design 

Pre-posttest, 2 group 

4 

633 

0.351-0.39, 1.10) 

94% 

.97 

posttest,2 group 

4 

232 

0.41 (-0.00, 0.82) 

40% 


Quality Newcastle-scale 

^4 points 

5 

493 

0.16 (-0.10,0.42) 

46% 

.03 

^3 points 

3 

372 

0.89(0.29, 1.49) 

70% 


Degree 

undergraduate 

3 

474 

0.37(-0.67, 1.41) 

95% 

.94 

graduate 

5 

391 

0.33 (0.07,0.58) 

28% 


Participant 

student 

4 

612 

0.31 (-0.46, 1.08) 

94% 

.77 

professionals 

4 

253 

0.43(0.15, 0.72) 

11% 


Duration 

£ 1 week 

5 

620 

0.51 (-0.10, 1.12) 

90% 

.26 

< 1 week 

2 

200 

0.05 (-0.23, 0.33) 

0% 


not mentioned 

1 

45 

0.471-0.20, 1.14) 

- 


Exercise 

present 

5 

582 

0.531-0.09, 1.15) 

90% 

.38 

absent 

3 

283 

0.171-0.34, 0.68) 

72% 


Discussion 

present 

4 

569 

0.68 (0.05, 1.32) 

91% 

.08 

absent 

4 

296 

0.071-0.18, 0.31) 

0% 


Repetition 

persistent access 

4 

582 

0.64(0.02, 1.26) 

91% 

.10 

single instance 

4 

283 

0.071-0.19, 0.33) 

5% 



In subgroup analyses, the effect sizes showed significantly positive number (favoring Internet-based 
learning) in studies of low quality score, delivering graduate degree courses, with public health professionals, 
the presence of discussion and ongoing access to learning materials (Table 3). 


3. CONCLUSION 

To our knowledge, this is the first meta-analysis to explore the effect of Internet-based public health training 
on participants’ knowledge improvement compared to no-intervention and traditional FTF format. The 
results of the present study revealed that Internet-based learning in public health training has a large positive 
effect compared with no-intervention. In contrast, the pooled effect sizes of Internet-based learning in 
comparison with traditional FTF instructional formats were small and of no statistical significance. The 
findings indicated that Internet-based learning is educationally beneficial and can achieve similar effects as 
that of traditional FTF instructional approach in public health training context, which concurred with 
previous reviews regarding Internet-based learning in other health branches such as clinical context (Cook et 
al. 2008), basic medicine (Lewis 2003), and nursing courses (Lahti et al. 2013). 

However, results suggested heterogeneity existed across studies. With reference to the preliminary 
evidence (Cook et al. 2010; Lahti et al. 2013), we hypothesized that the inconsistency might arise from the 
variations in study design, the involved education topics, participant type, and instructional design aspects of 
the retained studies. Yet the results of the subgroup analyses seemed only partially explaining such 
inconsistency. 

The present exploratory meta-analyses have several limitations. Firstly, we chose to collect only 
published articles in English, which could bring publication bias, despite there being no significant evidence 
of publication bias detected in meta-analysis of traditional FTF control group; Secondly, the sample size of 
1183 participants was relatively small, and the educational outcomes were restricted to knowledge level due 
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to limited quantitative data in other outcome levels such as skills and practice. Thirdly, unexplained 
inconsistencies would allow us to make tenuous inferences. 

In summary, despite conclusions could be weakened by the methodological limitations, heterogeneity 
across reviewed studies and the possibility of publication bias, meta-analyses of individual studies increased 
statistical power by reducing the standard error of the weighted average effect size. The synthesized evidence 
demonstrated that Internet-based learning appears to have a consistent positive effect as compared to 
no-intervention and have a similar effect to traditional FTF instruction methods in public health training 
context. Even the Internet-based learning is not superior to traditional FTF formats, it can, however, offer an 
alternative method of learning in public health training areas. In general, this review offers important 
information to increase knowledge about the effectiveness of the state-of-the-art education methods. 
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